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Enrolling in college and earning a degree or certificate is 
one of the primary pathways to economic success; how- 
ever, many students who begin a college education fail to 
complete it. In 2017, median weekly earnings of full-time 
workers with an associate degree was 17 percent higher than 
full-time workers with a high school diploma only.! Despite 
the economic advantages of educational attainment, only 32 
percent of students enrolled in two-year institutions com- 
plete their programs within three years, while 60 percent 
of students enrolled in four-year institutions complete their 
programs within six years. The first year of college is critical, 
as 38 percent and 19 percent of first-time, full-time students 
in two-year and four-year institutions, respectively, fail to 
return to the same institution for their second year.” As a 
result, several interventions have been designed to support 
student retention and degree completion among college 
students. One example is Success Boston Coaching. 


Success Boston Coaching is a coaching intervention for stu- 
dents who are traditionally underrepresented in college to 
help them transition from high school to college and prog- 
ress in college. Students are paired with a dedicated coach 


starting as early as the spring of their senior year of high 
school and receive coaching through their first two years in 
college. As Boston’s citywide college completion initiative, 
Success Boston partners with existing nonprofit organiza- 
tions focused on coaching and mentoring to deliver these 
one-on-one coaching services. Nonprofit coaching partners 
may also provide students with other direct services such as 
tutoring and career readiness support, and financial support 
that includes scholarships, transportation subsidies, and 
funding for school-related materials and supplies.* 


This What Works Clearinghouse (WWC) report, part of 

the WWC’s Supporting Postsecondary Success topic area, 
explores the effects of Success Boston Coaching on progress- 
ing in college and academic achievement. The WWC identi- 
fied seven studies of Success Boston Coaching. Two of these 
studies meet WWC standards.* The evidence presented in 
this report is from studies of the impact of Success Boston 
Coaching on high school and postsecondary students— 
including Asian, African-American, Hispanic, and White 
students—in a variety of school settings, including urban and 
suburban areas. 


What Happens When Students Participate in Success Boston Coaching?® 


The evidence indicates that implementing Success Boston 
Coaching: 


e may increase progression in college 
¢ may increase academic achievement 


Findings on Success Boston Coaching from two studies 
that meet WWC standards are shown in Table 1. The table 


reports an effectiveness rating, the improvement index, and 
the number of studies that contributed to the findings. The 
improvement index is a measure of the intervention’s effect 
on an outcome. It can be interpreted as the expected change 
in percentile rank for an average comparison group student 
if that student had received the intervention. 


Table 1. Summary of findings on Success Boston Coaching from studies that meet WWC Standards 


Study findings Evidence meeting WWC standards (version 4.0) 
Improvement index Number Number 
Outcome domain Effectiveness rating (percentile points) of studies of students 
Progressing in college Potentially positive effects +9 2 8,375 
Academic achievement Potentially positive effects +4 2 5,543 


Note: The improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the 


intervention. For example, an improvement index of +4 means that the expected percentile rank of the average comparison group student would increase by 4 points if the 
student received Success Boston Coaching. The improvement index values are generated by averaging findings from the outcome analyses that meet WWC standards, as 
reported by Linkow et al. (2017) and Linkow et al. (2019). A positive improvement index does not necessarily mean the estimated effect is statistically significant. Progressing 
in college outcomes reported in these studies include college persistence to the second year of higher education. Academic achievement outcomes reported in these studies 
include cumulative grade point average. The effects of Success Boston Coaching are not known for other outcomes within the Supporting Postsecondary Success topic area, 
including college enrollment, college attendance, postsecondary degree attainment, credential attainment, employment, and earings. 


BOX 1. HOW THE WWC REVIEWS AND DESCRIBES EVIDENCE 


The WWC evaluates evidence based on the quality and results of reviewed studies. The criteria the WWC uses for evaluating 
evidence are defined in the Procedures and Standards Handbooks and the Review Protocols. The studies summarized in this report 
were reviewed under WWC Standards (version 4.0) and the Supporting Postsecondary Success topic area protocol (version 4.0). 


To determine the effectiveness rating, the WWC considers what methods each study used, the direction of the effects, and the 
number of studies that tested the intervention. The higher the effectiveness rating, the more certain the WWC is about the reported 
results and about what will happen if the same intervention is implemented again. The following key explains the relationship between 


effectiveness ratings and the statements used in this report: 


Effectiveness rating Rating interpretation 


Description of the evidence 


Positive (or negative) effects The intervention is likely to change an Strong evidence of a positive effect, with no 


outcome 


overriding contrary evidence 


Potentially positive (or negative) effects The intervention may change an outcome Evidence of a positive effect with no overriding 


contrary evidence 


No discernible effects The intervention may result in little to no No affirmative evidence of effects 


change in an outcome 


Mixed effects The intervention has inconsistent effects Evidence includes studies in at least two of 


onan outcome 


How is Success Boston Coaching Implemented? 


The following section provides details of how Success Boston 
Coaching was implemented. This information can help edu- 
cators identify the requirements for implementing Success 
Boston Coaching and determine whether implementing this 
intervention would be feasible in their districts or schools. 
Information on Success Boston Coaching presented in this 
section comes from the studies that meet WWC standards 
(Linkow et al., 2017; Linkow et al., 2019), Success Boston’s 
website, and from correspondence with the developer. 


e¢ Goal: Success Boston Coaching aims to improve the 
college completion rates of Boston-area high school 
graduates by providing one-on-one transition coaching 
to students starting as early as their senior year in high 
school through their first two years in college. 


e Target population: Success Boston Coaching is designed 
for high school seniors in transition to attending two- 
and four-year colleges in the Greater Boston area who 
are traditionally underrepresented in college, such as 
low-income students, first-generation students, and 
students of color. Students are eligible to participate in 
Success Boston Coaching as early as the spring of their 
senior year of high school. 


e Method of delivery: Success Boston Coaching provides 
one-on-one transition coaching through nonprofit 
coaching partners that have included the Boston 
Private Industry Council, Bottom Line, College Bound 
Dorchester, Freedom House, Hyde Square Task Force, 


these categories: studies with positive effects, 
studies with negative effects, or more studies 
with indeterminate effects than with positive or 
negative effects 


Sociedad Latina, The Steppingstone Foundation, West 
End House Boys & Girls Club, and uAspire. These 
partners range from national college access and success 
organizations to local community-based organizations 
that serve populations traditionally underrepresented 
in college. Through in-person, phone, email, social 
media, and text communications, students receive 
sustained and proactive support from coaches staffed 
by the partner organizations. 


Frequency and duration of service: Students can 
receive coaching as early as the spring of their senior 
year of high school and coaching can continue through 
their first two years in college. Students’ one-on-one 
interactions with their coaches may range from three to 
15 times per year and last between 25 and 40 minutes. 


Intervention components: Success Boston Coaching 
is comprised of the following key components: one-on- 
one transition coaching, financial aid information and 
nudges, other direct services, financial support, and 
professional development for coaches. Refer to Table 2 
for additional details. 


Comparison group: In the two studies that contribute 
to this intervention report, students in the comparison 


group received traditional counseling supports 
provided by their high school and college. 


N 


Table 2. Components of Success Boston Coaching 


Key component Description 

One-on-one Students receive one-on-one transition coaching starting as early as the spring of their senior year in high school and coaching can 

transition continue through their first two years in college. Coaching is primarily conducted in-person, but students may reach their coaches 

coaching via phone, email, text message, or social media. Coaches meet with students as frequently as needed throughout their 
participation. As of the 2019-20 academic year, students meet with their coaches at least two times per month. Topics addressed 
during coaching sessions include academics, financial aid, career planning, and managing life responsibilities. Depending on the 
nonprofit coaching partner, coach caseloads range from 12 to 65 students; in the 2015-16 academic year, students’ one-on-one 
interactions with their coaches ranged from three to 15 per year and lasted between 25 and 40 minutes. 

Financial aid Success Boston Coaching partners with uAspire, a national nonprofit organization focused on increasing students’ knowledge of 


information and 
nudges 


Other direct 
services 


Financial 
support 


Professional 
development 
for coaches 


financial information and resources available to help make college affordable for students. uAspire supports students in completing 
financial aid forms. Additionally, uAspire administers a text messaging program for students during their first year of college to remind 
them of important financial aid deadlines and provide financial aid-related information and resources. Students can text back to the 
messages to receive help from uAspire staff. Students receive, on average, eight messages during their first year of college through 
the following summer. 


Coaching partners may offer tutoring for students. They may also host special events for students, such as career night presentations 
or mock job interviews with volunteer professionals from the community. 


Coaching partners may offer scholarships, transportation subsidies, or financial support for school-related materials, such as supplies. 


Coaching partners provide their coaches with training. The content and duration of the trainings vary based on organization. Coaches 
participate in Success Boston Coaching network-wide monthly meetings for continuing professional development and coaches can 
request specific training topics for these meetings. Example topics include: financial aid and financial literacy, coaching styles, building 
relationships with students, and bolstering student engagement. In addition, uAspire provides professional development for coaches 
on financial aid topics and processes. Some colleges hold ongoing monthly or biweekly meetings for all coaches on their campuses. 


What Does Success Boston Coaching Cost? 


The WWC identified a number of cost components from 2015. For the 2015-16 academic year-the first year in which 
the most recent study reviewed (Linkow et al., 2019) and the intervention was scaled from serving about 300 students 
the Success Boston website. This preliminary list of costs is to 1,000 students per cohort annually—the estimated total 
not designed to be exhaustive; rather, it provides educators cost was $5,762 per student, which accounts for both the 

an overview of the major resources needed to implement scale-up expenses and the administrative costs to fulfill 
Success Boston Coaching. The intervention costs described in Social Innovation Fund (SIF) grant reporting requirements. 
Table 3 are based on the information available as of Without the additional administrative costs associated with 
March 2020. program scale-up, in the 2016-17 academic year, the cost per 


Success Boston Coaching was awarded a Social Innovation 
Fund (SIF) grant to scale up coaching services in the fall of 


student decreased to $3,037. Costs may vary depending on 
the size and type of organization delivering the coaching. 


Table 3. Cost ingredients for Success Boston Coaching 


Costingredients Description Source of funding 

Personnel Success Boston Coaching is staffed by coaches from nonprofit coaching organizations who The Boston Foundation (TBF) 
provide direct services to students; program administrators to support the program and with support from a Social 
conduct recruitment activities; and college liaisons to coordinate campus coaching activities. Innovation Fund (SIF) grant from 
Success Boston Coaching also contracts with uAspire to provide staff who deliver financial the Corporation for National and 
aid training for coaches, FAFSA completion events for students, and a text messaging Community Service (CNCS); 
program for students focused on sharing financial aid-related information and resources. coaching partner organizations 

Facilities Facilities costs include space for administrative staff and for providing services. Success Coaching partner organizations; 
Boston Coaching relies on partner organizations to provide space for service delivery. Some __ colleges 
partners rely on dedicated campus space such as cubicles, scheduled time in offices, or 
classrooms provided by the colleges. 

Equipment and —_ Standard information technology (IT) equipment such as computers, phone, printers, and Coaching partner organizations; 

materials photocopiers for staff is required. colleges 

Direct Students may receive some direct cash or cash-equivalent assistance such as reduced-fare Coaching partner organizations; 

assistance public transportation passes, scholarships, and school supplies. transit authority 

Other Depending on the partner organizations, students may also receive additional supports such Coaching partner organizations; 


as tutoring, career night presentations or mock job interviews, and annual events. volunteers; AmeriCorps members 


Note: The Boston Foundation (TBF) provides approximately $1 million per year to fund coaching and other activities for transition coaching, such as training and supports 
for coaches and program management. Between the fall of 2015 and the fall of 2019, Success Boston received funding from a Social Innovation Fund (SIF) grant from the 
Corporation for National and Community Service (CNCS). The SIF grant required a match from The Boston Foundation and a local match from the nonprofit coaching partners. 


For More Information: 


About Success Boston Coaching 
The Boston Foundation 
75 Arlington Street, 3rd Floor 
Boston, MA 02116 


Attn: Antoniya Marinova, Assistant Director, Education to Career 


Email: Antoniya.Marinova@tbf.org Web: www.successboston.org Phone: (617) 338-3928 


About the cost of the intervention 
Web: www.successboston.org 


See also: 


Linkow, T., Bumgarner, E., Didriksen, H., Lack, K., Nichols, A., Dastrup, E., Dastrup, S., & Gamse, B. (2019). The story of scaling up: 
Interim report on the impact of Success Boston’s Coaching for Completion. Cambridge, MA: Abt Associates. Retrieved from 


https://eric.ed.gov/?id=ED602748 


Research Summary 


The WWC identified seven studies that investigated the effec- 
tiveness of Success Boston Coaching (Figure 1): 


¢ 2studies meet WWC group design standards with 
reservations 


¢ 1study does not meet WWC group design standards 
¢ 4studies are ineligible for review 


The WWC reviews findings on the intervention’s effects on 
eligible outcome domains from studies that meet stan- 
dards, either with or without reservations. Based on this 
review, the WWC generates an effectiveness rating, which 
summarizes how the intervention impacts, or changes, a 
particular outcome domain. The WWC reports additional 
supplemental findings, such as semesters enrolled full-time 
for progressing in college (Linkow et al., 2019), on the WWC 
website (https://whatworks.ed.gov). 


These supplemental findings and findings from studies that 
either do not meet WWC standards or are ineligible for 
review do not contribute to the effectiveness ratings. 


The two studies of Success Boston Coaching that meet WWC 
group design standards with reservations reported find- 
ings on progressing in college and academic achievement. 
No other findings in the studies meet WWC group design 
standards within any outcome domain included in the 
Supporting Postsecondary Success topic area.® Citations for 
the three studies reviewed for this report are listed in the 
References section, which begins on page 10. Citations for 
the four studies that are ineligible for review and the reasons 
the WWC determined they were ineligible are also listed in 
the References section. 


Figure 1. Effectiveness ratings for Success Boston Coaching 
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Success Boston Coaching has potentially positive effects on progressing in college 


The WWC determined that two studies that meet WWC group design standards with reservations (Linkow et al., 2017; 
Linkow et al., 2019) showed evidence of a positive and statistically significant effect of Success Boston Coaching on 


progressing in college. 


Success Boston Coaching has potentially positive effects on academic achievement 


The WWC determined that one study that meets WWC group design standards with reservations showed evidence of a 
positive and statistically significant effect of Success Boston Coaching on academic achievement (Linkow et al., 2017), 
and one study that meets WWC group design standards with reservations showed evidence of an indeterminate effect of 
Success Boston Coaching on academic achievement (Linkow et al., 2019). 


Main Findings 


Table 4 shows the findings from the two Success Boston 
Coaching studies that meet WWC standards. The table 
includes WWC calculations of the mean difference, effect 


size, and performance of the intervention group relative to 
the comparison group. Based on findings from the two stud- 


ies that meet WWC standards, the effectiveness rating for 


progressing in college is potentially positive effects, indicating 


Table 4. Findings by outcome domain from studies of Success Boston Coaching that meet WWC 


evidence of a positive effect with no overriding contrary 
evidence. These findings are based on 8,375 students. The 
effectiveness rating for academic achievement is potentially 
positive effects, indicating evidence of a positive effect with 
no overriding contrary evidence. These findings are based 


on 5,543 students. 


Standards 
Mean 
(standard deviation) WWC calculations 
Sample Intervention Comparison Mean Effect Improvement 
Measure (study) Study sample size group group difference size index p-value 
College persistence Full Sample 2,912 83.00 75.00 8.00 0.30 +12 <.05 
- enrolled ina 
second year of 
higher education (%) 
(Linkow et al., 2017)? 
College persistence Full Sample 5,863 82.60 78.30 4.30 0.17 +7 <.01 
- enrolled ina 
second year of 
higher education (%) 
(Linkow et al., 2019)’ 
Outcome average for progressing in college across all studies 0.23 +9 
Cumulative grade Full Sample 1,990 2.45 2.26 0.19 0.18 +7 <.01 
point average (0.92) (1.08) 
(Linkow et al., 2017)? 
Cumulative grade Full Sample 3,553 2.37 2.36 0.01 0.01 0 0.82 
point average (1.03) (1.10) 
(Linkow et al., 2019)’ 
Outcome average for academic achievement 0.10 +4 


Notes: For mean difference and effect size values reported in the table, a positive number favors the intervention group and a negative number favors the comparison group. 
The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are given the 
intervention (measured in standard deviations of the outcome measure). An indicator of the effect of the intervention, the improvement index can be interpreted as the expected 
change in percentile rank for an average comparison group student if that student had received the intervention. For example, an improvement index of +4 means that the 
expected percentile rank of the average comparison group student would increase by 4 points if the student received Success Boston Coaching. A positive improvement index 
does not necessarily mean the estimated effect is statistically significant. Some statistics may not sum as expected due to rounding. 
2 Linkow et al. (2017) did not require corrections for clustering or multiple comparisons nor difference-in-differences adjustments. Standard deviations reported for the cumulative 
grade point average outcome were provided by the author in response to a WWC author query. This study is characterized as having a statistically significant positive effect on 
progressing in college and academic achievement because the estimated effects are positive and statistically significant. 
> Linkow et al. (2019) did not require corrections for clustering or multiple comparisons nor difference-in-differences adjustments. Standard deviations reported for the cumulative 
grade point average outcome were provided by the author in response to a WWC author query. This study is characterized as having a statistically significant positive effect 

on progressing in college because the estimated effect is positive and statistically significant. This study is characterized as having an indeterminate effect on academic 


achievement because the estimated effect is not statistically significant. 


For more information, please refer to the WWC Procedures Handbook, version 4.0, page 22. 


In What Context Was Success Boston Coaching Studied? 


The following section provides information on the setting of 
the two studies of Success Boston Coaching that meet WWC 
standards, and a description of the participants in the research. 


WHERE THE STUDY WAS CONDUCTED 


oe 


Race 


Asian White Unspecified 


African 
American 


Free & reduced- 
price lunch: 
77% 


Special 
education: 
11% 


English 
learners: 
11% 


This information can help educators understand the context in 
which the studies of Success Boston Coaching were conducted, 
and determine whether the intervention might be suitable for 
their setting. 


2 studies, 8,375 students in Massachusetts 


Districts: Urban and suburban school districts 


Gender 


Female Male 


Grades: 10 1 


© © 


12-Postsecondary 


Details of Each Study that Meets WWC Standards 


This section presents details for the studies of Success Boston 
Coaching that meet WWC standards. These details include 
the full study reference, findings description, findings 
summary, and description of study characteristics. A sum- 
mary of domain findings for each study is presented below, 
followed by a description of the study characteristics. These 
study-level details include contextual information about the 
study setting, methods, sample, intervention group, com- 
parison group, outcomes, and implementation details. For 
additional information, readers should refer to the original 
studies. 


Research details for Linkow et al. (2017) 


Linkow, T., Gamse, B., Unlu, F., Bumgarner, E., Didriksen, 
H., Furey, J., Meneses, M., Sami, M., Nichols, A. (2017). The 


power of coaching: Interim report on the impact of Success Bos- 
ton’s transition coaching on college success. Cambridge, MA: 

Abt Associates. Retrieved from https://www.tbf.org/-/media/ 
tbforg/files/reports/sb-317-interim-outcomes-report-final.pdf 


Findings from Linkow et al. (2017) show evidence of a statis- 
tically significant positive effect of Success Boston Coaching in 
the progressing in college domain (Table 5). This finding is 
based on an outcome analysis that includes 2,512 students. 
The study showed evidence of a statistically significant 
positive effect in the academic achievement domain. This 
finding is based on an outcome analysis that includes 1,990 
students. The findings and research details summarized 

for this study come from three related reports, including 
the primary study listed above. See the References section, 
which begins on page 10, for a list of all related publications. 


Table 5. Summary of findings from Linkow et al. (2017) 


Meets WWC Group Design Standards With Reservations 


Study findings 
; Sample Average Improvement Statistically 
Outcome domain size effect size index significant 
Progressing in college 2,912 students 0.30 +12 Yes 
Academic achievement 1,990 students 0.18 +7 Yes 


Table 6. Description of study characteristics for Linkow et al. (2017) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
group 


Comparison 
group 


Outcomes and 
measurement 


Additional 
implementation 
details 


Meets WWC Group Design Standards With Reservations. This is a quasi-experimental design (QED) with baseline 
equivalence established on the analytic sample.’ For more information on how the WWC assigns study ratings, please see 
the WWC Procedures and Standards Handbooks (version 4.0) and WWC Standards Briefs, available on the WWC website. 


Success Boston Coaching was implemented in the Boston, Massachusetts metro area. Partners included The Boston 
Foundation, the City of Boston, Boston Public Schools, the University of Massachusetts Boston, Bunker Hill Community 
College, other regional colleges and universities, uAspire, the Boston Private Industry Council, and other local nonprofit 
organizations. During the 2013-14 and 2014-15 academic years, students received coaching from seven nonprofit 
organizations: American Student Assistance, Boston Private Industry Council, Bottom Line, Freedom House, Hyde Square 
Task Force, Sociedad Latina, and West End House. A national nonprofit organization, uAspire, delivered financial aid 
advising to students and professional development for coaches. Students attended nine partner colleges: Benjamin Franklin 
Institute of Technology, Bridgewater State University, Bunker Hill Community College, Massachusetts Bay Community 
College, Northeastern University, Roxbury Community College, Salem State University, Suffolk University, and University of 
Massachusetts Boston. 


Students in Success Boston Coaching were statistically matched to students who did not participate in the coaching based 
on the following characteristics: gender, race/ethnicity, high school academic achievement, and socioeconomic status. The 
matching process limited the Success Boston Coaching students and comparison students to similar locales, high school 
environments, and college settings. 


The study included two cohorts of students who graduated from Boston Public Schools in 2013 and 2014. Students who 
graduated in 2013 were followed for two years and those who graduated in 2014 were followed for three years to measure 
the effect of Success Boston Coaching. The primary analyses were limited to a two-year follow-up period from high school 
graduation to standardize the amount of time elapsed for both the 2013 and 2014 high school graduating classes. 


Students who met the following three criteria were included in the study sample; students had to: (1) enroll in college in the 
fall after high school graduation, (2) enroll in a college with at least one Success Boston Coaching student and at least one 
potential comparison student enrolled that year, and (3) have no missing information on key baseline characteristics used in 
the matching process. The analytic sample for the persistence in college outcome, as measured by enrollment in a second 
year of college, included 678 students in the intervention group and 1,834 students in the comparison group. The analytic 
sample for the academic achievement outcome, as measured by cumulative grade point average in college, included 610 
students in the intervention group and 1,380 students in the comparison group. 


The study sample was comprised of 42 percent African-American students, 37 percent Hispanic students, 12 percent 
Asian students, and eight percent White students. Sixty-one percent of the study sample was female, 15 percent were 
English learners, 10 percent had an Individualized Education Plan, and 85 percent received free or reduced price lunch. 


Students in Success Boston Coaching began receiving one-on-one coaching from experienced counselors starting as early 
as the end of high school and continuing through the first two years of college. Coaching was available on demand to help 
students navigate the college process while developing their sense of agency, autonomy, and independence. The coaching 
was designed to develop students’ life, study, help-seeking, and academic skills and helped students hone their relationship, 
goal setting, and networking skills. Job and career mentoring were also provided. Coaching activities began after high 
school graduation for 92 percent of students. 


Students in the comparison group did not participate in Success Boston Coaching but were eligible to receive traditional 
counseling services while in high school and college. 


Study authors reported findings on two outcome measures that are eligible for review under the Supporting Postsecondary 
Success topic area. These findings include enrollment in a second year of higher education (progressing in college domain) 
and cumulative grade p oint average (academic achievement domain). 


The study also reported supplemental findings for a subsample of students in the progressing in college domain, including 
enrollment in a third year of higher education. Summaries of these findings are available on the WWC website 
(httos://whatworks.ed.gov). The supplemental findings do not factor into the intervention’s rating of effectiveness. 


Other findings were included in the study that do not meet WWC group design standards. These findings include enrolled 
in a second year of higher education, enrolled in a third year of higher education, and cumulative grade point average, 
reported separately by underrepresented minority status (i.e., Black, Native American, Hispanic) or not traditionally 
underrepresented (i.e., Asian, White). These findings do not meet WWC group design standards because the study authors 
used a quasi-experimental design in which the analytic intervention and comparison groups do not satisfy the baseline 
equivalence requirement. 


The Boston Foundation oversaw the Success Boston Coaching network that facilitates communication across the initiative. 
The network also provided coaches access to specialized training about financial aid from uAspire, a national nonprofit 
organization, as well as access to training on other topics. 


Research details for Linkow et al. (2019) on an outcome analysis that includes 5,863 students. The 
finding on the academic achievement domain, which shows 
evidence of an indeterminate effect, is based on an outcome 
analysis that includes 3,553 students. The findings and 
research details summarized for this study come from three 
related reports, including the primary study listed to the 
left. See the References section, which begins on page 10, for 
Findings from Linkow et al. (2019) show evidence of a statis- a list of all related publications. 

tically significant positive effect of Success Boston Coaching 

on progressing in college (Table 7). This finding is based 


Linkow, T., Bumgarner, E., Didriksen, H., Lack, K., Nichols, 
A., Dastrup, E., Dastrup, S., & Gamse, B. (2019). The story 
of scaling up: Interim report on the impact of Success Boston’s 
Coaching for Completion. Cambridge, MA: Abt Associates. 
Retrieved from https://eric.ed.gov/?id=ED602748 


Table 7. Summary of findings from Linkow et al. (2019) 


Meets WWC Group Design Standards With Reservations 


Study findings 
Sample Average Improvement Statistically 
Outcome domain size effect size index significant 
Progressing in college 5,863 students 0.17 +7 Yes 
Academic achievement 3,553 students 0.01 0 No 


Table 8. Description of study characteristics for Linkow et al. (2019) 


WWC evidence Meets WWC Group Design Standards With Reservations. This is a quasi-experimental design (QED) with baseline 
rating equivalence established on the analytic sample.® 


Setting Success Boston Coaching was implemented in the Boston, Massachusetts metro area. Partners included The Boston 
Foundation, the City of Boston, Boston Public Schools, the University of Massachusetts Boston, Bunker Hill Community 
College, other regional colleges and universities, uAspire, the Boston Private Industry Council, and other local nonprofit 
organizations. During the 2015-16 and 2016-17 academic year, students included in the study received coaching 
from eight nonprofit coaching organizations: Boston Private Industry Council, Bottom Line, College Bound Dorchester, 
Freedom House, Hyde Square Task Force, Sociedad Latina, the Steppingstone Foundation, and West End House. A 
national nonprofit, uAspire, delivered financial aid advising to students and professional development for coaches. The 
majority of students (74 percent) attended one of eleven colleges: Benjamin Franklin Institute of Technology, Bridgewater 
State University, Bunker Hill Community College, Framingham State University, Massachusetts Bay Community 
College, Northeastern University, Roxbury Community College, Salem State University, Suffolk University, University of 
Massachusetts Boston, and Wentworth Institute of Technology. 


Methods The study included two cohorts of students who graduated from Boston Public Schools in 2015 and 2016 and entered 
college in the fall of 2015 and 2016. Outcomes were measured after students in each cohort would have been out of high 
school for three and two years, respectively. Students participating in Success Boston Coaching were identified using 
the program’s administrative database. Students who met the following three criteria were included in the study sample; 
students had to: (1) enroll in college in the fall after high school graduation; (2) enroll in a college in which at least one 
Success Boston Coaching student and at least one potential comparison student were enrolled that year; and (3) have no 
missing information on key baseline characteristics used in the matching process. The authors used a propensity score 
matching procedure that blocked the matching process by unique combinations of high school graduating cohorts and the 
postsecondary institutions they attended. Each Success Boston Coaching student was matched to one or more comparison 
students in his or her block who shared similar baseline characteristics, based on a caliper of 0.4 standard deviations of the 
propensity score. 


The analytic sample for the persistence in college outcome, as measured by enrollment in the second year of higher 
education, was comprised of 1,234 students in the intervention group and 4,629 students in the comparison group. The 
analytic sample for the academic achievement outcome, as measured by student's cumulative grade point average in 
college, included 814 students in the intervention group and 2,739 students in the comparison group. 


Study sample The study sample was comprised of 42 percent African-American students, 31 percent Hispanic students, 17 percent 
Asian students, and eight percent White students. On average, students were 18 years old at baseline. Sixty percent of the 
sample were female, 10 percent were English learners, 11 percent had an Individualized Education Plan, and 74 percent 
received free or reduced-price lunch. 


Table 8. Description of study characteristics for Linkow et al. (2019) (continued) 


Intervention 
group 


Comparison 
group 


Outcomes and 
measurement 


Additional 
implementation 
details 


Nonprofit coaching organizations recruited Boston Public School graduates who were transitioning to college, and partnered 
with local colleges to coordinate coaching for students who qualify and enroll in Success Boston Coaching. Students and 
coaches connected in-person both on and off-campus using text messaging, email, or phone. Throughout the academic 
year, coaches provided students with ongoing, one-on-one support on academic, financial, career, and personal topics. For 
example, coaches offered on-demand guidance to help prepare students to navigate the college environment and become 
increasingly independent. Coaches also provided one-on-one support on life skills, study skills, help-seeking strategies, and 
academic skills. They helped students develop meaningful relationships, clarify goals, access networks, and understand 
college culture. Coaches also provided job and career mentoring as needed. Throughout the intervention, partner colleges 
communicated with coaches and helped coordinate coaching services on their campuses. The Success Boston Coaching 
network, which The Boston Foundation oversaw, facilitated communication across organizations and provided coaches 
access to specialized training about financial aid from uAspire, a national nonprofit organization, as well as access to 
training on other topics. 


Students in the comparison group had access to traditional college support services but not the coaching and support 
offered by the eight nonprofit organizations involved in Success Boston Coaching. 


Study authors reported findings on two outcome measures that are eligible for review under the Supporting Postsecondary 
Success topic area. These findings include enrollment in a second year of higher education (progressing in college domain) 
and cumulative grade point average (academic achievement domain). 


The study also reported supplemental findings in the progressing in college domain for a subsample of students, including 
semesters enrolled full-time and enrolled in a third year of higher education. Summaries of these findings are available 

on the WWC website (https://whatworks.ed.gov). The supplemental findings do not factor into the intervention’s rating of 
effectiveness. 


Other findings were included in the study that do not meet WWC group design standards. These findings include college 
credits earned, as well as subgroup findings for students who are female, male, African-American, initially enrolled in 
two-year college, and initially enrolled in four-year college for enrollment into the second and third year of college. These 
findings do not meet WWC group design standards because the study authors used a quasi-experimental design in which 
the analytic intervention and comparison groups do not satisfy the baseline equivalence requirement. 


The Boston Foundation oversaw the Success Boston Coaching network that facilitates communication across the initiative. 
The network also provided coaches access to specialized training about financial aid from uAspire, a national nonprofit 
organization, as well as access to training on other topics. 
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studies were reviewed by staff members from a different or- 
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studies because sample members and data collection and 
analysis procedures differed. There is no overlap in sample 
members between the two studies: Linkow et al. (2017) in- 
cludes students who graduated from high school in 2013 or 
2014, and Linkow et al. (2019) includes students who grad- 
uated from high school in 2015 or 2016. There are also dif- 
ferences in data collection and analysis procedures: Linkow 
et al. (2019) matches intervention and comparison students 
on baseline characteristics drawn from administrative data 
from the Massachusetts Department of Elementary and 
Secondary Education (MA DESE) only, while Linkow et al. 
(2017) matches intervention and comparison students on 
baseline characteristics drawn from a merge of administra- 
tive data from Boston Public Schools and MA DESE. 

5 The literature search reflects documents publicly available 
by March 2020. Reviews of the studies in this report used 
the standards from the WWC Procedures and Standards 
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point average outcome. These baseline measures were 
used as covariates in the analysis. Intervention students 
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radius matching method with replacement. Comparison 
students who were matched to multiple intervention stu- 
dents were weighted in the analysis to reflect the number 
of matches. 
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sample using data supplied by the author in response to 
a WWC author query. The author provided baseline high 
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status to establish baseline equivalence on the persistence 
outcome. The author provided high school grade point av- 
erage to establish baseline equivalence on the college grade 
point average outcome. These baseline measures were 
used as covariates in the analysis. Intervention students 
were matched to potential comparison students using a 
radius matching method with replacement. Comparison 
students who were matched to multiple intervention stu- 
dents were weighted in the analysis to reflect the number 
of matches. 


Recommended Citation 


What Works Clearinghouse, Institute of Education 
Sciences, U.S. Department of Education. (2020, October). 
Success Boston Coaching. Retrieved from 
https://whatworks.ed.gov. 


11 


